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Gap: A Difference-Education Intervention Improves First-Generation 
Students’ Academic Performance and All Students’ College Transition ” 1 

The findings from this review do not reflect the full body of research evidence 
on difference-education interventions. 


What is this study about? 

The study investigated the impact of attending a 
moderated panel discussion for incoming fresh- 
men on their adjustment to college. The panel 
featured demographically diverse college seniors 
who responded to questions about their experience 
of and adjustment to college. The study was con- 
ducted in a highly selective, midsized private univer- 
sity in the United States. 

All incoming first-generation college students— in 
this study, students whose parents do not have 
4-year college degrees— and a sample of incoming 
non-first-generation college students were invited to 
participate in the study. The two groups of students 
did not differ significantly by race or ethnicity. Of the 
168 students included in the study, 84 were ran- 
domly assigned to the intervention group and 84 to 
the comparison group. 

Incoming students in both groups attended one of 
eight moderated panel discussions, all featuring the 
same panel of eight demographically diverse col- 
lege seniors (three were first generation, five were 
non-first generation). Panelists were instructed to 
respond to questions differently depending upon the 
group of students in attendance. For the students 
in the intervention group, the panelists’ responses 
illustrated how their social class backgrounds both 
positively and negatively shaped their college expe- 
riences and influenced the strategies they adopted 


for success in college. For students in the compari- 
son group, the panelists’ stories included general 
content and did not highlight the students’ differ- 
ent backgrounds. After the panel, all students were 
invited to complete a survey and create a video 
testimonial about the panel’s main teachings. 

The study assessed the impact of the panelists’ 
stories by comparing grade point averages (GPAs) 
at the end of students’ freshman year. The authors 
were particularly interested in whether the interven- 
tion affected first-generation and non-first-genera- 
tion students differently. 2 


Features of a Difference-Education Intervention 


The difference-education intervention exposed 
incoming freshmen to a moderated panel 
discussion, during which a panel of diverse senior 
college students responded to questions about 
how their social-class backgrounds affected their 
experience of and adjustment to college. 
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What did the study find? 

The study assessed the impact of the difference- 
education intervention on GPAs at the end of fresh- 
man year. The study authors found, and the WWC 
confirmed, that there was a statistically significant 
effect on mean GPA: the intervention group had a 
higher mean GPA (3.47) than the comparison group 
(3.30). This effect was mostly due to the statisti- 
cally significant differences between the GPAs of 
first-generation students in the intervention and 
comparison groups. Specifically, the mean GPA for 
first-generation college students in the intervention 
group was higher than the mean GPA for first-gener- 
ation students in the comparison condition (3.47 vs. 
3.17, respectively); this difference was statistically 
significant. In contrast, the GPAs for non-first-gen- 
eration students were similar across the two groups 
(3.47 and 3.43, a difference that is not statistically 
significant). 


WWC Rating 


The research described in this 
report meets WWC group design 
standards without reservations 

This study is a well-executed randomized 
experiment with low attrition. 
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Appendix A: Study details 

Stephens, N. M., Hamedani, M. G., & Destin, M. (2014). Closing the social-class achievement gap: A dif- 
ference-education intervention improves first-generation students’ academic performance and all 
students’ college transition. Psychological Science, 25(4), 943-953. doi: 10.1177/0956797613518349 


Setting 

The study was conducted at a highly selective, midsized private university in the United 


States. 

Study sample 

The sample consisted of 168 incoming freshmen. Eighty-four students were randomly 
assigned to the intervention condition and 84 to the comparison condition. Students were 
recruited in two ways, before randomization. All incoming first-generation students— that is, 
students whose parents do not have 4-year college degrees— were identified and asked to 
participate in the study. The authors then identified and recruited non-first-generation students 
to match the racial/ethnic profile of the entire non-first-generation student population. 

The analytic sample consisted of the 147 students who attended the panel and completed the 
survey at the end of the year. Of the 147, 66 were first generation; of these 45% were White, 
24% Latino, 17% Asian or Asian American, and 14% African American. Fifty-nine percent 
were low income (defined as eligible to receive Pell grants). Eighty-one students in the ana- 
lytic sample were non-first generation, of whom 52% were White, 25% Asian, 15% Latino, 

7% African American, and 1 % Native American. Nine percent of these students qualified as 
low income. Seven participants did not consent to have their grades accessed and thus were 
excluded from the analyses of academic performance. Additionally, six GPA outliers were 
excluded from analyses of academic performance. These 13 exclusions resulted in an analytic 
sample of 134 students for the academic achievement outcome; 66 of these students were in 
the intervention group and 68 in the comparison group. 

Intervention 

group 

Students in the intervention group attended one of eight moderated panel discussions, all featur- 
ing the same panel of eight demographically diverse college seniors (three were first genera- 
tion, five were non-first generation). Each panel discussion lasted 1 hour, during which panelists 
responded to questions posed by a moderator. In this condition, panelists talked about how 
their backgrounds affected their college experiences. Their responses illustrated how their social 
class backgrounds both positively and negatively shaped their college experiences and influ- 
enced the strategies they adopted for success in college. At the end of the session, attendees 
were invited to take a short survey and make a video testimonial on the lessons they learned 
from the panel. At the end of their freshman year, students completed another survey. 

Comparison 

group 

Students in the comparison group also attended one of eight moderated panel discussions, 
all featuring the same panel of eight demographically diverse college seniors (three were 
first generation, five were non-first generation). Each panel discussion lasted 1 hour, during 
which panelists responded to questions posed by a moderator. In this condition, panelists’ 
responses included general content and did not highlight the students’ different backgrounds. 
At the end of the session, attendees were invited to take a short survey and make a video 
testimonial on the lessons they learned from the panel. At the end of their freshman year, stu- 
dents completed another survey. 
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Outcomes and 
measurement 


Support for 
implementation 

Reason for 
review 


The study had one primary outcome: grade point average (GPA) at the end of the freshman 
year. GPA was measured using first-year cumulative GPAs from official registrar records. For a 
more detailed description of this outcome measure, see Appendix B. 

The authors provide information on how the moderators introduced the study to the two dif- 
ferent conditions and the questions that were used in each condition. They provide samples of 
the ways students answered questions in the two conditions. 

This study was identified for review by receiving media attention. 
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Appendix B: Outcome measures for the academic achievement domain 


Academic achievement 


Grade point average (GPA) This outcome is based on data from student transcripts received from the university. A mean GPA was mea- 

sured using first-year cumulative GPAs from official registrar records. 

Table Notes: This single-study review does not include other outcomes reported by the authors, including appreciation of difference and perspective taking, taking advantage of 
college resources, and psychosocial outcomes (i.e., stress and anxiety, psychological adjustment, academic engagement, social engagement). These outcomes were not eligible 
for review under the Review Protocol for Individual Studies in the Postsecondary Education topic area, version 2.0. 


October 2014 


Page 5 



WWC Single Study Review 


Appendix C: Study findings for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Academic achievement 

Grade point average (GPA) 

College 

freshmen 

134 

3.47 

(0.38) 

3.30 

(0.39) 

0.17 

0.44 

+17 

.006 

Domain average for GPA 






0.44 

+17 

Statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average individual’s percentile rank that can be expected if the individual is given the intervention. 

Study Notes: Adjusted mean GPAs, standard deviations, and analytic sample sizes were obtained through an author query and are not included in the published study. No cor- 
rections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were reported in the original study. This 
study is characterized as having a statistically significant positive effect because the effect for at least one measure within the domain is positive and statistically significant, and 
no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to the WWC Standards and Procedures Handbook 
(version 3.0), pp. 25-26. 
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Appendix D: Supplemental findings for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Academic achievement 

Grade point average 

First- 

generation 

freshmen 

60 

3.47 (0.36) 

3.17(0.32) 

0.30 

0.87 

+31 

<.001 

Grade point average 

Non first- 

74 

3.47 (0.39) 

3.43 (0.39) 

0.04 

0.10 

+4 

.66 


generation 

freshmen 


Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is 
a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individuals who are given the intervention (mea- 
sured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average individual’s per- 
centile rank that can be expected if the individual is given the intervention. First-generation freshmen include those students whose parents do not have 4-year college degrees. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were reported in the 
original study. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC group design standards. The review reports the WWC’s assessment of 
whether the study meets WWC group design standards and summarizes the study findings following WWC conventions for report- 
ing evidence on effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 2.0. A 
quick review of this study was released in March 2014, and this report is the follow-up review that replaces that initial assessment. The 
WWC rating applies only to the study outcomes that were eligible for review under this topic area. The reported analyses in this single 
study review are only for those eligible outcomes that either met WWC group design standards without reservations or met WWC 
group design standards with reservations, and do not necessarily apply to all results presented in the study. 

2 There were seven outcomes included in the study that are not described in this WWC report. These include appreciation of differ- 
ence and perspective taking, taking advantage of college resources, and psychosocial outcomes (i.e., stress and anxiety, psychologi- 
cal adjustment, academic engagement, social engagement). See the table notes in Appendix B for more information. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, October). 

WWC review of the report: Closing the social-class achievement gap: A difference-education intervention 
improves first-generation students’ academic performance and all students’ college transition. Retrieved from 
http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual attributable to the intervention. As the average individual 
starts at the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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